Robust Elastic Net Regression

We propose a robust elastic net (REN) model for high-dimensional sparse regression and give its performance guarantees (both the statistical error bound and the optimization bound). A simple idea of trimming the inner product is applied to the elastic net model. Specifically, we robustify the covariance matrix by trimming the inner product based on the intuition that the trimmed inner product can not be significant affected by a bounded number of arbitrarily corrupted points (outliers). The REN model can also derive two interesting special cases: robust Lasso and robust soft thresholding. Comprehensive experimental results show that the robustness of the proposed model consistently outperforms the original elastic net and matches the performance guarantees nicely. Introduction Over the past decades, sparse linear regression has been and is still one of the most powerful tools in statistics and machine learning. It seeks to represent a response variable as a sparse linear combination of covariates. Recently, sparse regression has received much attention and found interesting applications in cases where the number of variables p is far greater than the number of observationsn. The regressor in high-dimensional regime tends to be sparse or near sparse, which guarantees the high-dimensional signal can be efficiently recovered despite the underdetermined nature of the problem [3, 7]. However, data corruption is very common in highdimensional big data. Research has demonstrated the current sparse linear regression (e.g. Lasso) performs poorly when handling dirty data [6]. Therefore how to robustify the sparse linear regression becomes a major concern that draws increasingly more attentions. Robust sparse linear regression can be roughly categorized into several lines of researches. One is to first remove the detected outliers and then perform the regression. However, outlier removal is not suitable for the high-dimensional regime, because outliers might not exhibit any strangeness in the ambient space due to the high-dimensional noise [22]. Another type of approaches include replacing the standard mean squared loss with a more robust loss function such as trimmed loss and median squared loss. Such approaches usually can not give performance guarantees. Methods [12, 15, 13] have been developed to handle arbitrary corruption in the response variable, but fail with corrupted covariates. [14] proposes a robust Lasso that considers the stochastic noise or small bounded noise in the covariates. [5] also considers similar corruption settings and proposes a robust OMP algorithm for Lasso. For the same noise, [17] proposes the matrix uncertainty selector that serves as a robust estimator.